Tesla recalls cars with EMMC faults, calling them "wear parts" Hacker Day

2021-11-13 07:16:48 By : Ms. Alice Chan

For anyone who spends a lot of time playing Raspberry Pi, this is a familiar problem-over time, the flash memory in the SD card reaches its write cycle limit and leads to a series of confusing before it fails completely mistake. Although flash memory is fast, compact and mechanically reliable, its writable life is always much shorter than that of magnetic technology.

Of course, these problems can be successfully alleviated through proper wear-leveling techniques and careful use. Surprisingly, when a large car manufacturer fails to achieve these basic functions, it is like several Tesla models. As the car’s Linux operating system records too much of its 8 GB eMMC storage, the flash memory module has worn out. This can cause widespread failures in the car, usually putting it into limp mode and disabling many functions controlled via the touch screen.

As the problem affected important subsystems such as heaters, defrosters, and warning systems, NHTSA wrote to the automaker in January requesting a recall. Tesla's response acquiesced to this request with some consternation, and played down the seriousness of the problem. Now they claim that the eMMC chip, the ball grid is soldered to the motherboard, cannot be accessed without disassembling the dashboard, and is not specifically mentioned in the user manual, should be regarded as a "wear item", so it should not be subject to this review.

Historically, the main electronic components in cars were not considered consumables. Although it is not uncommon for some cars to experience problems with engine control units or body control modules, they are generally not considered wear parts that require regular replacement. So far, precedents have regarded these components as components that can last the life of the vehicle and replaced them in the event of unexpected failures. Tesla’s situation is different, eMMC failure is largely inevitable. As expected by accidental manufacturing defects, this is not an isolated failure in a small number of cars, but a problem that affects every car that goes offline on a specific date. In some construction months, the failure rate was as high as 30%. Since computers and touch screens are responsible for many important vehicle functions, end users will not easily ignore this defect.

Tesla's assertion that eMMC chips should be considered "wearables" is suspicious at best. It is true that flash memory does wear out, as Tesla pointed out when discussing technical limitations. Many parts in modern cars wear out over time-brake pads, belts and air filters are common examples. The difference is that these parts are designed to be replaced by the end user or a typical mechanic.

Attempting to claim that the ball grid array chip permanently soldered to the PCB and buried in the dashboard is a wear item is obviously absurd. If so, we would like to see a few things. There will be a recommended time and mileage. Based on these times and mileage, the eMMC will be changed to avoid unexpected failures, which will be listed in the manual. In addition, Tesla's repair process will involve removing the eMMC chip from the board and replacing it directly. Given that Tesla is replacing the computer as a whole, this indicates that this part will not be considered a wearable item by anyone anywhere.

Obviously, the chip can be replaced, but this is not an easy task. After removing the computer motherboard from the car, it must be backed up and stored via JTAG. Then, it must be carefully reflowed to remove the chip. This is a very delicate process that is likely to damage other components on the board. If the chip is a wear part, no special BGA reflow equipment is needed to replace it. We will see Tesla doing this often, replacing chips with less than $7 instead of replacing the entire motherboard instead of spending thousands of dollars. It is true that certain parts of modern cars are also time-consuming to replace-such as timing belts, water pumps, etc. However, in these cases, the car manufacturer once again made it clear that these are wear items, develop maintenance plans for them, and develop standard procedures to change them.

Every time the brakes wear out, no one will bear to replace the entire front suspension setup-the car manufacturer realizes that the brake pads are worn items and designs accordingly. Tesla just lost the ball, and the frequency of writing to the flash memory is too high to be easy to replace. The correct solution is trivial. Either stop recording a lot of records to flash memory, or make it easier to swap out.

Maybe put the log in its own partition. Although SD cards may not meet the requirements for storing car operating systems, they can be used as an inexpensive place to store non-critical logs that may never be read. Or, put the eMMC chip on a removable module, or just use M.2 drives with automotive-grade connectors.

The issue is said to only affect models built before March 2018, which run on NVIDIA Tegra 3. Later models are based on Intel Atom and have a larger onboard eMMC chip. These modules have not yet exhibited the same failure, and Tesla claims that they should not have encountered this problem. We will wait and see.

As a Tesla owner, I know this will be a problem. I saw it before I bought Model Y. I can't understand why they don't use a movable flash. Even with the advancement of technology, packaging will change. But at least there are adapters that can solve this problem. In this case, it is much more complicated than replacing the flash on the iPhone, which most consumers cannot do now.

I'm not sure if this is a problem with Model Y and the newer Teslas. But I am also not sure that making it movable is a good idea. Over time, removable flash memory may fail. A better solution might be to use better software techniques to reduce logging and handle flash writes appropriately, and use (pseudo) single-level cell flash chips for the main subsystem. Even with high writing and better software, these can last for decades and basically never wear out (well, the PCB itself may end up... maybe 50-100 years?). However, entertainment storage should be removable.

No, the correct way is not to let important functions such as the defroster depend on the eMMC chip (or computer). This is why I think Tesla does not belong to the automotive industry. Because they have shown time and time again that they do not understand safety engineering, IMHO, they have shown great contempt for their customers.

It is fine if they do not write logs to eMMC. Simply move the log to a removable flash drive to solve the problem. I would even say that there are 2 SD cards-one for the user profile and one for the log. This will keep the eMMC in a read-only state and avoid the whole prank they find themselves in.

Flash memory also has the chance of reading interference and corrupting data anyway, so even if you do zero recording, data corruption will still make your windshield wiper inoperable. This is a completely avoidable design choice.

Not the point. Safety functions like defrost should not fail for any of these reasons. The SD card or Emmc should be able to fail without disabling the safety features

There is almost zero reason to buy a car that can record your every micro-action and record your face.

Can you integrate the panorama into your life more deeply? Literally, every minute of the day feels like a colonoscopy.

"No, the correct way is not to rely on the eMMC chip (or computer) for important functions such as the defroster."

Why the defroster needs more than a simple switch or relay is beyond my understanding.

The defrost heater needs a bit of intelligence, because you don't want it to be on forever, but a 555 timer or similar device should be able to do the job.

Most cars now control the power supply of the heater so that it does not necessarily consume power when the battery is low, which may be part of the problem.

If the battery state of charge is <70%, my gasoline engine Ford won't let my heated windshield work, and also disables the start/stop function, although strangely it still allows my heated seats and steering wheel to warm up.

It is not that the defroster is particularly malfunctioning, but that the entire computer, which is confusingly called "MCU", cannot be booted from a bad flash memory chip. NHTSA can only force a recall for safety reasons. A) depends on the Web GUI displayed on the MCU and B) is a defroster among the functions that are technically a safety item.

It's like saying "Microsoft must prevent *applications* from being maliciously launched by vulnerabilities", but regulators did not comment on the behavior of third-party applications, so they replaced "applications with binary files such as calc.exe or Notepad" program" . Executable program".

> Because you don't want it to leave forever

In winter, yes, you do. Once you turn on the windshield blower to clear the frost, it will condense and fog the rear window. Every time you drive, you almost turn on the defroster so that the moisture will stay in the air and be blown away. Otherwise you will start to accumulate water in the car, because people's shoes and breathing, in the spring, your foot pads will drip.

Oh, yes, I forgot that Tesla owners have heated garages.

> In winter, yes. Once you turn on the windshield blower to clear the frost, it will condense and fog the rear window. Every time you drive, you almost turn on the defroster so that the moisture will stay in the air and be blown away. Otherwise you will start to accumulate water in the car, because people's shoes and breathing, in the spring, your foot pads will drip.

Not sure what car you drive, but all my cars since the 90s have automatically turned off the heated windshield and rear window defogger in a few minutes, and I rarely need to use them again after that. In fact, I can't remember any car in the UK today that does not turn off in a short period of time. Due to the temperature of -5 degrees Celsius, I took two 35-mile trips today after the ice was initially cleared.

I use AC without recirculation, although the car disables recirculation anyway when the humidity sensor thinks it is too humid to recirculate.

Maybe my passengers and I breathe less, haha, if the mat is too wet, I often put the mat in it to dry.

As long as you breathe in enough fresh air, it shouldn’t be a problem, unless your car is full of wet passengers, which may mean you have to turn it on every five or ten minutes.

I am currently driving a Japanese car in the mid-90s. The smartest function of the defroster is that the lights in the dashboard will glow to remind you that it is still running.

> Due to the temperature of -5 degrees Celsius.

For three weeks, it has been alternating between -10 and -25 C. If you don't always turn on the rear window heater, it will turn white due to the moisture breathed by car passengers.

> I often put my mat inside to dry

Mine are fixed to the floor and cannot be removed. The rubber foot guards on the top are detachable, but obviously they will not absorb water. Some people put old newspapers under rubber mats and around the footwells to absorb moisture, but water also sticks to the door panels and seats, so there is always a problem. Unless you can keep the car warm enough for long enough to completely dry (ie in a heated garage), you must keep defrosting and heating running at full speed every time you drive.

The problem is that it takes a few hours for the car to dry out normally, but even then, you can usually only drive for 30-45 minutes at most, which means you can only heat it up enough for the water to start to evaporate, and then you stop and the water again Frozen on the surface, every time you step into the car, you will add more moisture to it.

Although this year I did buy a small pillow-sized bag of silicone beads and put it on the front window under the vent. This helps a lot—you just need to bring it in and bake it in the oven occasionally—but I still need to open the rear window to defrost to prevent fogging when I drive.

> I am driving a Japanese car from the mid-90s

I have never done it myself, and what annoys me is that they put the indicator on the wrong side compared to all the other cars I had to jump in, and I ended up turning on the wipers at random.

The last car I owned that was not defrosted in time was the original British Mini, not the BMW version, and it will time out after a period of time. For the past 20-25 years, I have owned Citroen, Volvo, Ford, Audi, BMW and Vauxhall cars, all of which have timers.

I know that my son Vauxhall before I discovered that it did make a huge difference many years ago, I cleaned his windows properly for him. The steam in his windows was terrible. My Volvo was always steaming until I sent it to the valet parking because of milk overflow. I was surprised by the changes it made, so I now keep the internal windows clean.

> It has been alternating between -10 and -25 C for three weeks.

Fortunately, we will not see these temperatures in central England.

>>I often put my mat inside to dry>My mat is fixed on the floor and cannot be removed

I always get the manufacturer's mats, which can be loosened to remove, and because they have rubber non-slip mats, the carpet underneath will stay dry when wet.

>I use AC without recirculation

Mine has three manual settings: outside air, inside air and 50/50. In fact, when it snows, the vent below the windshield will be blocked by ice, and any air that the vent can take in through the gap comes from the engine compartment. In any case, you basically have to switch the ventilation to internal air to heat it up, but once the engine gets hot and the vents start to clear, you will start to get fresh air-although of course the air becomes humid due to the melting ice.

-5 C is nothing. You hardly even need to scrape the windows. I ride a bicycle and work at -5 C. In contrast, for durations of -15 C and below, you need to scrape all windows from the outside and inside every morning.

This is why LADA is the perfect winter car. It leaks severely, cannot keep any moist air inside, and is so inefficient that you can get several kilowatts of heating power from the cabin radiator.

> In the past 20-25 years, I have owned Citroen, Volvo, Ford, Audi, BMW and Vauxhall cars, and they all have timers.

interesting. They have to make them unique in the British market, because in any car I see here, they don't have any timers. You press the button and the heater turns on until you press the button again. It cannot be said that in the last ten years, it may have some temperature sensors to control it, but at least since the 90s and early 00s, it is actually just a switch and a relay-nothing more.

On my old car (2007), there is a whole section of the manual dedicated to explaining the working principle of the defroster. There is a booster that can increase the voltage to 90V to provide 1500W (or other) power. It monitors the car voltage, outside temperature, windshield temperature and has a timer. If the circuit is open, it should report an error code. I have described about 50% of the functions. I want to know how many 555s are needed.

@Dude: The (rear) defroster has no effect on the accumulation of water (condensation). It just prevents condensation on the rear window. To dry the interior of the car, you need to use dry air to carry the water out of the car.

What really makes me unbelievable is that every engineer seems to know the write limit of flash memory.

Even if I perform low-level programming on atmel chips, I know that emmc and eeprom have write restrictions. If it is used as some kind of watchdog timer ram, that is, the chip wakes up, checks eeprom, writes eeprom after calibration or recalibration, the watchdog timer hits, the chip checks eeprom, and sees an incorrect shutdown flag. Within the time limit, pull out the last calibration. The rated write of eeprom is 100K, so considering the large number of pre-circulation writes and the 1KB eeprom size, you can kill the chip fairly quickly.

Flash may only have 1000 cycles. I took this into consideration in another project.

It is more likely that Tesla’s culture is so emphasized that it has not considered these issues.

It is hard to say that the development team may fall into a very annoying political mode, some developers refuse to accept criticism, and because of seniority will merge garbage code.

When Elon says "jump", you jump-otherwise you will be fired.

Maybe the system-level design assumes that the logs will be recorded to tmpfs and then uploaded to the cloud... This may be their negligence in being written to flash memory. Maybe they are written to flash backed tmpfs...

We already know that Tesla is recording a lot of data from telemetry... so they really have no reason to record anything to flash.

100k is not a limit, it is just a guarantee from Atmel that it will actually handle more writes. Some of you may remember that when it was live, I think it reached a few million writes before it failed.

I heard that this is an error in the log settings. An indicator (I want to say external temperature, but I'm not sure) should be recorded every ten minutes or so, but somehow it is set to 0.1 seconds. Or at least this is my memory of rumors. So yes, they knew the write limit in flash memory and made sure it wouldn't be a problem, but then someone messed up the log setting and broke it anyway.

You will be surprised by the numbers that are not there. I had to ask about the wear limits of the hardware we bought because I knew it, but it took the manufacturer almost a month to find someone in the company who recognized these limits. In our case, we need a service life of 3 years, and the part (at our usage rate) is rated for 7 years. So that's it.

As early as 1992 when I was working in the defense industry, even at that time, we knew the limitations of flash memory, wrote a wear leveling algorithm, and designed our hardware in a way that made flash memory replaceable.

> What really makes me unbelievable is that every engineer seems to know the write limit of flash memory.

The hardware designer may be very clear but not aware of the logging that will be performed. Software programmers may never see the hardware or understand the type of memory they will log in.

It may be that non-engineers think it would be good to record so much information without knowing the limit of available memory.

It reminds me of the designer placing the oil filter in a location where it needs to be partially dropped from the engine to reach them. They may design an excellent engine, but may not be able to see how it will be installed in the car.

This may boil down to people making decisions outside of their area of ​​expertise: hardware engineers blindly use improperly written software libraries, and software engineers sign up to use flash memory without understanding the consequences.

I totally agree. I think they have some great ideas, but I don't believe they will take the necessary time to fully review their designs. My main problem is that their autopilot function will kill people. In the engineer’s creed, it says that one of our jobs is to "protect the public." fail. This shows me that they either don't have good engineers or their marketing pushes things out of the door before they are ready. Either way, they have cultural (business) issues that they cannot trust.

Consider vibration, but anti-vibration flash interconnects can also be used. But the path length will be longer, but this is easy to fix.. oh. Hope this coverage encourages them to put in 128Gb chips, and still intentionally use a small amount of space and the wear level algorithm that balances the rewrite cycle

this. They need to take advantage of economies of scale and only use larger chips. If they did this for the first time, they would not spend a lot of money on repairing a small flash memory chip in thousands of cars. Oops.

I challenge you to say that this is more complicated than replacing the flash on the iPhone! Compared with mobile phones, Tesla looks very simple! Have you tried to disassemble the iPhone and reassemble it without damaging the screen? iPhone also has TINY circuit.

It might be easier to turn on the iPhone with the car dashboard completely removed...

Buying a new iPhone is cheaper than taking apart the dashboard and then putting it all back.

Well, flash wear, why not "wear"? Wear leveling is something that most products include, regardless of whether flash memory will not last forever. Tesla is like any car company. The first adopter is the one who can find all the mistakes. The first generation cars are usually full of them.

The fact of the eventual wear is not disputed. What is disputed is that the chip was intended to be replaced. If they only write to memory in rare cases, then it can easily last millions of miles of driving.

Can't the board be changed? Does this person have to throw away their car? I think the "replaceable chip" situation is often not-disposable items. The chip is technically replaceable, but the cost may exceed the value of the circuit board. From a business point of view, this does not make sense. Yes, this is a design flaw. Yes, Tesla should fix the design and the cars. Don't hold your breath for the individually replaceable board-level components. For many reasons, this will not happen.

I think the key is that by calling it "wear and tear items," Tesla implies that wear and tear is the expected behavior. However, because they did not make it replaceable, the entire multi-$K motherboard must be replaced when the chip fails. I believe what Gravis said is that it is hard to believe that the chip will wear out, not that the chip should be replaced separately.

yes, I agree. "Wear item" is a way to avoid warranty. They should repair people's cars and redesign them to avoid this problem. Maybe renovation is the way to go. Allowing individuals to solve the problem is not feasible for most people. Hack-aday readers are happy to insert things, but grandma is not.

I think of Honda in the early 1990s (?). Their timing belt started to break after driving 80,000 miles. So, their recollection is: update the user manual saying...change the timing belt every 60K miles

(Nevertheless, I do own a Honda)

There is also the fact that everyone else in the automotive industry is smart enough not to hit their flash memory with log file writes in production.

Everyone else in the industry is using devices with much less rated write cycles. There is no problem of reaching these limits. There is a simple rule:

Do not write to flash memory in production unless it is a last resort.

The difference is that all other automakers know that too much logging on eMMC is a bad idea. I work in the automotive industry, where we use modern operating systems and usually record many things on multi-core CPUs, and we accurately calculate how much we did and how much we could write before the eMMC died. We can say how many hours our equipment can run without causing any damage. It's just a beginner's fault that Tesla didn't do this calculation.

"Now they claim that the eMMC chip, the ball grid is soldered to the motherboard, cannot be accessed without disassembling the dashboard, and it is not specifically mentioned in the user manual, which should be regarded as a "wear item" and therefore should not be subject to this restricted review. "

It looks very much in line with the way Tesla and other technology companies operate. Obviously, there is actually nothing that can be "wrongly made"...

An important way to deal with flash memory chips without "wear and tear" becoming a big problem is to use other things for actual "daily" activation, such as some battery-backed RAM, and then write only when our main power supply disappears Flash memory.

For an electric car equipped with a 100 kWh battery, the small memory that runs for a few months should be insignificant... After all, give it a super capacitor. If the battery suddenly disappears, it can quickly save its contents. flash. In this setting, the wear of the flashlight is usually less than the wear of the car. (After all, this is fairly common in many other applications, where flash memory is used in high-wear environments, such as battery-backed disk caches on some RAID cards.)

However, if you want to treat the Flash chip as a wear part, you should not solder it to the board unless it is a daughter board. (That is, the storage should be in the socket/connector.)

"The issue is said to only affect models built before March 2018, which run on NVIDIA Tegra 3. Later models are based on Intel Atom and are equipped with larger eMMC chips. These modules have not yet shown the same Failure, Tesla claimed that they should not have encountered this problem. We will wait and see."

It doesn't matter if the eMMC chip is bigger, it will still wear out. The only exception is if it is so large that the time it takes to wear becomes similar to the expected life of the car. But even so, people usually hope that the device will not be unusable because of the damage of relatively cheap things, so the eMMC flash memory should be located on the connector anyway. (I personally don’t like an SBC with eMMC flash memory. For me, it’s just electronic waste waiting to happen in the future. Why throw away the entire SBC just because the flash memory is dead, even though we can usually start other things here. ...)

Although I like Lewin Day’s suggestion to store logs and other non-important data on a separate storage device, this storage device is much simpler.

It *really* important is how big the flash unit is. For the same write throughput and technology, larger flash memory cells will last longer.

I recommend reading the entire argument, not just the initial part of it.

"The only exception is if it is so large that the time it takes to wear becomes similar to the expected life of the car."

Now, cars that do have eMMC problems have 8GB of flash memory, and some of them will have problems after a relatively short period of time.

Doubling the number of Flash will only double the failure time, nothing more.

But as I have already said, it can only be repaired when it is expected to continue through the life cycle of the product.

Although some people may regard a car as something that only needs 5 years of service life, if one pays more attention to environmental protection and economy, people will hope that its service life is at least 4-5 times longer than this.

Just turn off the memory, so I may be wrong, but I seem to remember that the log only takes up a small part of 8GB, and the rest are system files. If they (for example) use 500MB for logs, then increasing to 16GB would be a 32-fold improvement.

Your math is a bit wrong, if it is 500MB log, 7.5GB system file, on 16GB it can be 7.5GB system file, 8.5GB log, so 17x

The flash memory is erased in blocks, and the wear leveling algorithm rearranges the blocks, so updating a single byte may cause anything from 4 KB to 0.5 megabytes to be written on the flash memory. This is called "write magnification". This is why it is usually a silly idea to save logs on a flash memory device-repeatedly adding a few bytes to a file will cause wear leveling and garbage collection algorithms to destroy the chip over time.

> Doubling the number of Flash will only double the failure time

Not necessarily, if it uses MLC technology and larger block sizes to increase capacity. See "Write Amplification".

Yes, but the write amplification is basically the same in both cases. But yes, write amplification will greatly shorten the life expectancy of flash memory.

That is, you can quickly enable the drive to withstand more than 100 terabytes of writes during its lifetime, because only a few hundred gigabytes of actual data are stuffed into it.

Logging is such an application, it adds hundreds of bytes at a time from time to time... Each time the shuffled data is much more than the actual content written, just to find space for it.

One of the best reasons for battery-backed RAM drives for logging is a great thing.

If you double the capacity by doubling the level of each cell, then at best you will not have any effect on the wear behavior, because each write affects the same number of cells. In the worst case, you make the situation worse because MLC is more fragile.

Reaching higher charge levels generally makes the flash memory cell more sensitive to long-term drift. In other words, it forgets faster.

Since in each write/erase cycle, we weaken the insulation, so that the charge stored in the floating gate has a slightly more conductive path to escape. However, for new batteries stored at room temperature, the time frame we are discussing is from several years to decades. But technically speaking, there is no "wear and tear" threshold, but the data retention time is reduced to a level that is not practical for the application.

Yes, everyone who puts a USB thumb drive into the time capsule is a "good guy". Since the data retention time of high-density flash memory in flash memory in the early 2000s is usually about 5-8 years. But after using the drive for a period of time, the retention time may drop to within a few months.

(However, if you ask Wikipedia about flash memory, the data retention time in 85 C is about 30 years or more, because Microchip claims its low density, high data retention rate, and industrial flash memory that is usually used in its microcontrollers. That is, , It’s not even economical even in enterprise SSDs.)

In the end, increasing capacity by moving to a higher degree of MLC is as wise as sticking to the same solution.

On the other hand, increasing the capacity by increasing the number of flash memory cells is a feasible solution. However, this is only a solution, provided that it can actually last the life of a larger product or can be easily replaced in the field. (The BGA solder package on the system board is not at all.)

However, my personal thoughts on SLC, "MLC", TLC, QLC, and finally PLC flash memory. Why don't we call it "DLC" which stands for double-layer battery. After all, "MLC" stands for multi-level battery, which is technically applicable to all non-SLC. But technically, it should probably be DLC (2), QLC (4), OLC (8), HLC (16), etc., because this is the actual number of bits stored in each level to encode Number of levels, but to be fair, I am pedantic here. Nonetheless, it does show that just adding 1 bit to the cell increases the sensitivity, because of the sudden need for twice the actual threshold level.

"Obviously, there is actually nothing that can be "wrongly made"..."

You got it wrong.

Yes, this is an unforeseen problem that put Tesla into trouble. All they can do is work hard to get out of the predicament-hope they can get some recourse from the BGA manufacturer? ? This is a matter of time and labor. Tesla should do a time and movement optimization. There will be many different modules. They can set up a central repair station to repair each type of module, and reserve a batch of inventory to ship to dozens of service bureaus everywhere. They can then schedule the car for repairs using the repair module known at hand to minimize tasks. You can use a well-trained team to do the module. No matter how it is made, it will be very popular-they must eat. All they can do is to minimize it and provide a lender if possible. A sad accident. Sockets will alleviate this situation-if it is predictable? It has been suggested that this could have been expected and it could have been avoided as the way Linux addresses/refreshes/wear levels changes.

Flash wear is not an unknown quantity: any professional electronic design that includes it should (and in general) analyze the expected life of flash memory. This is easy to do when you know the write rate of flash memory. For almost all embedded automation systems, this may be well controlled (Tesla claims that the write endurance of its flash memory is in full compliance with industry standards: this is true, but only half of the equation. Most automation systems will not Continuously write to flash memory, of course, will not be at the rate that Tesla does). Using embedded linux is a bit difficult (a naive method like the Raspberry Pi is not good here) because you are not sure how much data will be written, but it is easy to alleviate, for example, equip your device with a separate partition or even a chip root The file system (which is strictly kept read-only except for firmware upgrades), your configuration options (usually with very low write rates), and your logs. Then make sure that even if the log partition fails, your system can still run.

It is not clear whether Tesla has made any preliminary assessments of the lifespan of flash memory, but it is certain that they later intensified the wear and tear of the chip when they developed the software and introduced it to the car: as their firmware became The larger and larger, the space available for wear leveling on the flash memory is getting smaller and smaller, the amount of recorded data increases, and the lifespan is significantly shortened. If the software remains the same, it may not be a problem. Obviously, they either did not pay attention to the problem or felt that it was not important enough to spend any energy to solve the problem. They also neglected to ensure that even if the flash memory becomes unwritable, the system can still work.

This is not an unforeseen problem, in fact anyone with experience in embedded linux should have considered it.

I totally agree with you. Since the moment I realized that flash memory wears out, it has become a consideration for any use of flash storage that is owned or built. Even if it is just choosing an SD card for some trivial ultra-non-critical data storage, there will be a little whisper in the back of my head telling me to remember flash wear. Engineers don't think this is both shocking and frustrating.

Engineers may have considered it, but as with "autonomous driving" systems, anyone who filed a complaint could be fired for "performance reasons."

seriously. Forbes had an article that day in which they interviewed an original engineer behind Autopilot. He tried to raise concerns, but was basically fired for speaking out. That was before the first wave of autonomous driving deaths hit the media.

"It is not clear whether Tesla has made any preliminary assessments of the lifespan of flash memory, but it is certain that they later intensified the wear and tear of the chip when they developed the software and introduced it to the car: as their firmware became Bigger and bigger, the space available for wear leveling on the flash memory is getting smaller and smaller, the amount of recorded data increases, and the lifespan is significantly shortened."

Want to know if the mobile phone will encounter this situation as the program becomes larger and larger?

The first reason to replace the previous generation mobile phone is that Android will crash when the flash memory on the main chip of the mobile phone is used up. You cannot put everything on the SD card. For some reason, even 16 GB is almost used up these days.

Please say it again? Maybe at some point in history. Today is the battery. It is clearly designed as a failure and difficult to repair, so it is mandatory to replace it.

BS, which is MFgrs, refuses to update the software. Mainly because of software growth, the memory/resources of the old unit are limited, therefore, there is no update. Buy a new suction cup. .

It's the battery. Assuming you get about 1000 charges/discharges and do this every night, your battery will be about three years old. This is why they make it difficult for them to change. They want you to buy a new phone every 3 years, regardless of whether it still works (except for the battery). So it is as bad as Tesla...

>Today is the battery.

I did not find this problem because I always buy mobile phones with replaceable batteries.

> BS, which is MFgrs, refuses to update the software.

do not know. I still got the latest update. My problem is that even if I don’t put any photos or videos on the phone, but on the SD card, I only have 2.3 GB of 16 GB available. I changed everything as much as possible, but it is still getting tighter and soon I will have to start deleting the application to make it work.

The point is, when I list the installed apps, the sizes they list are completely ridiculous (obviously wrong), such as 64.1 MB. The system claims that I have installed 3.6 GB of applications, but I cannot list applications over 600 MB on the system. Even if this is true, why does a normal operating system take up more than 10 GB of space?

All geeks here should work for Tesla and earn millions every year.

When you have a CEO who promises twice but fulfills half of it each time, the only people who can stay in the company are those who are good at disguising their abilities. People who are truly capable will be fired because they will say "no" to stupid or unrealistic demands.

People do predict that the entire "touch screen in the middle controls everything" will cause electronic equipment to malfunction, malfunction or become obsolete during the life of the car.

"Every time the brakes wear out, no one will stand to replace the entire front suspension setup."

No, but don't let automakers do the same stupid things. The lower control arm bushing on my Civic is cast in the control arm. Want to replace the bushing? Replace the entire part. I can understand why they do this, but it is a waste.

On my 2013 Ford Escape, replace the battery.

Step 1. Remove the windshield wiper:

https://youtu.be/-lJiRmC9igc

I started thinking about replacing the battery on the '14 Escape, but I did not give up and let the dealer replace it during regular maintenance.

Have you tried adding coolant to the reservoir?

The Turbo on my '16 Renault Trafic is malfunctioning. When I removed the exhaust pipe, subframe, steering rack and DPF, I didn't know. I can reach Turbo at this time, but I hope I don't.

When I had a Chevrolet HHR, replacing the headlight bulb meant removing the front wheel and removing the plastic cover around the wheel cover. After the first time, I decided to cut a square on the plastic cover and put a small hinged door in its place so that in the future I can turn the wheel and reach in to replace it.

Want to know whether after-sales parts correctly deal with these oversights?

This is one of the reasons why I no longer buy Chevrolet. They work well and I really like everything I have, but they make basic maintenance too difficult.

A few years ago, we had a fully loaded BMW 3 Series as a test car. The light bulb of "Angel Eyes" is broken. If I remember correctly, the dealer reported a lot of money and about 4-8 hours of work. We say BS and consider replacing it:-Remove the wheel-Remove the mudguard (about 15 self-tapping screws)-Remove the front bumper, be careful of those parking sensors... We didn't have it immediately, and then took it to another The dealer is there. It turned out that there was a small hole in the fender through which the child's hand could pass. Obviously they have someone who can do these jobs.

This is not a problem for most cars, but this car is completely optional and has the largest engine or other things.

Replace the battery in '14 Escape-it takes about 2 hours because you have to remove the air filter assembly (and various other parts) or (easier) the window cover to remove the battery...

Therefore, if you are a Canadian and started to worry about your battery in Escape in 2013, please get CAA... They will replace your battery at the price of the battery. Tip; if you want to keep this car for 5 years, just Get their best. A one-year membership fee of $74 may sound cheaper than the store price. I don't know if this also applies to AAA.

Look at modern motorcycle engines. The cylinder assembly is the fixed part of the upper casting of the engine. The cylinder itself is Nikasil-coated aluminum, which is also the fixed part of the upper engine cover. Cannot be replaced. If the cylinders wear out, you can drill them and repaint them about 3 times (buy larger pistons each time). But after that, all that's left is to throw away the entire engine block.

With an off-road vehicle and a cylinder, I think it's not bad. Only one cylinder needs to be drilled and only one piston needs to be replaced. But modern four-cylinder motorcycles have such engines.

The metaphor is not good. Unrebuildable engines make the most sense for street motorcycles. Compared with passenger car engines, motorcycle engines bear much less wear and unfavorable conditions, and most motorcycle owners have never shortened the service life of the engine.

The lower control arm is ridiculously expensive in terms of labor replacement, which is why I recently had to get rid of my beloved 2007 ES350 at 200,000 miles, otherwise its mechanical shape is very good. Lexus cars are reliable, but if something happens, good luck.

In the first company I worked for, our products were based on multiple SBCs (including RPi1). We did use SD cards, but except for system updates, they are completely read-only. Building such a system with Buildroot is not difficult, but not entirely easy, because some GNU/Linux components like to non-optionally need to access variable storage, even if you use tmpfs, you must ensure that it does not run out of memory...

This is the most annoying aspect of the whole thing: almost everyone who has been serious about SBC development knows not to fall into this trap. "You will run out of your flash!" is almost as common as "3D printing is not food safe!"

It is inexcusable not to disturb the correct implementation of the design at all.

The main problem with eMMC is the early cars. I don’t completely blame them (although they shouldn’t be so angry about it now *now*), because I also installed a bunch of embedded Linux systems that use flash memory, and eventually had to replace their flash memory for the same reason. unit. At that time (ten years ago), people's understanding of it (I mean the broader IT/linux community) was not as important as it is now to handle logging, etc. correctly.

However, they don’t have to make replacement so difficult...

Using that standard, we will have no cars, no computers, no houses, and no nuts. Human beings are not perfect, we will not do perfect things, otherwise expectation is stupid.

It's like building a house without a circuit breaker box. We will replace all the wiring while frying.

This may be the age-old but very simple question where the people of E.Engineers and Software do not sing in the same hymn-software humans are moving forward, wow, we have so many sensors and so much data, we should always put them all Record it to help our autopilot (maybe), or maybe to prove how durable our brake pads are, etc. But the construction and design of the system assumed a less crazy record level, much longer than the expected service life of the car through software updates.

Cloud, cloud, cloud, and put in some ubiquitous 5G. Solve one's log problem.

Hush, don't give them ideas!

> We do use SD cards, but except for system updates, they are completely read-only. It is not difficult to build such a system with Buildroot

I only say one thing: systemd and its companions (hostnamed, etc.). I don't know how to set the hostname on the RPI for network boot. Some systemd components just *want* to access the file system, you cannot use tmpfs to disguise them.

Devuan doesn't use systemd, just says...

F*** systemd and Lennart P. Use rusty spoons. "This is just an initialization system! The other components are optional! Until they are not, but ours is better, farmer!" "Your use case is unreasonable!"

Of course, they can cache any writes to flash memory in RAM until a complete power failure event occurs? Even if there is an unexpected power failure, you can store enough energy in the capacitor to write the changes to the flash memory before the last watt of juice is exhausted?

This requires a vision that they seem to miss.

That being said, when someone points out a problem, the engineer is likely to say "everything will be fine". Or when the engineer points out the problem, the manager says "everything will be fine".

(I pointed out the problem in the product and was told "it will be fine")

When the upper-level managers are a group of hackers who try to "cheat" their ways and try their best to make things happen, the lower-level engineers start to only do what they need to do, nothing more.

When the management does not want to listen to what the engineer says, the engineer must play dumb and ignore all support functions and "scaffolding", which are actually required for the job. In other words, they cracked the new feature request, and the product became technically fragile-bodges on bodges.

can not agree with you more. As the owner of the 2016 Model S, I just realized the flashlight problem because my device started to have problems. I have been a software engineer for decades; many years ago, the problem of cache writing to ram and updating flash memory only became a simple problem during major events such as power failure.

It should be pointed out that the Raspberry Pi can now use a third-party Argon One M.2 chassis, which adds functions such as a power button, and the most important thing is to be able to actually operate the M.2 SATA SSD compatible style drive. Accept any size M.2 SATA SSD with Key-B or Key-B&M. It actually uses USB 3, so it's also faster.

Simply using a micro SD as the main "hard drive" for Raspberry Pi or Tesla is not a good idea for long-term use. Even if it is a medium quality micro SD card. Has taken several breaks in various ways. This is not exactly what they are experiencing now.

If you use flash memory/eMMC or anything similar for storage, your software must take into account that you can only rewrite a limited number of cells before they start to fail. Memory cache, bad sector management, evenly distribute writes on all sectors, and not much content is written at the beginning.

This was also a big problem when I was working at TomTom many years ago. All of these are resolved by changing the content and method of writing to flash memory.

That's it. Everyone knows this, but Tesla means "Go fast and break things..." is a good strategy for car production. Hope this will cost a lot of money, because they know this is a bad idea...

Just want to know how hot you can get before they fall off the board or start to move. Whether to use higher temperature solder because of the car. As we all know, flash memory can be re-annealed by heating it to 250 degrees Celsius for "a few hours". I think this is too high, but would like to know if it is a dose/time issue, for example heating it to 170C and leaving it at 170C overnight may have an impact. The goal is that you can make a clip with a heater that can maintain a temperature of 170C, then bend your arm to the dashboard with it, clamp the edge of the circuit board on the chip, insert and leave it. It may be much cheaper repair.

Considering that the common reflow temperature peaks of lead and RoHS solders are around 240C/260C, annealing is sufficient to desolder them.

It is tricky to make reliable vibration-proof connectors and also have to deal with large temperature cycles. Don't expect to use a commercial-grade socket without going through a shaking and baking cycle. That is, forget the usual flash memory cards, etc.

They can use the adapter pcb to solder it to the board, just like making any esp module. So the problem now is the soldering iron and not the reflow station.

There are many armchair novices who will jump to the "outlet" as a solution to all problems without realizing the mechanical and/or electrical aspects of the problem. Seen too much on the Internet these days.

For any incidental liability, it needs to be done by the factory authorized party, because Tesla is in trouble.

As long as the authorized repairer has the correct equipment and replacement parts, it doesn't matter how they install it. First of all, Tesla must admit that there is a problem before finding a solution...

By the way, your PCBs will turn brown and delaminate because they cannot withstand high temperatures for a long time.

PCB glass point (Tg) classification:

Low Tg: about 130°C Medium Tg: >=150°C High Tg: >=170°C

Um? Tg is not suitable for cured epoxy...

You missed a mile...

> For PCB, Tg corresponds to the temperature at which the glass fiber becomes amorphous during the lamination process under the high temperature and pressure of different material layers. It is not the maximum operating temperature of the PCB, but the short time that the PCBA can withstand before it deteriorates.

>If the working temperature of the PCB exceeds its Tg for a long time, the PCB will change from the glass state to the rubber state, and its performance will be affected. Tg guarantees the mechanical stability and normal operation of the PCB during its service life.

>Tg is one of the key characteristics to be considered when specifying a PCB. It is very important to determine the exposure temperature of the PCB in the early stages of the design to select the right material, especially for PCBs that are exposed to high operating temperatures.

This may be a stupid question, but why can't the firmware and hardware be updated during the TSB event? Update the firmware to write to the SSD on the daughter board via JTAG? As long as the wiring is long enough, the daughter board can be installed in a more accessible location.

Why bother with JTAG? JTAG is a test interface, not a universal bus, or flash memory. A microcontroller/board with a JTAG interface means that it is a slave device and cannot switch JTAG pins.

eMMC is a wearable item, but Tesla's architecture is destroying it. This is similar to having a partially stuck brake on every car, except in this case, the brakes should be several million miles, but the defect will burn them suddenly and without warning within tens of thousands of miles. The recall requires replacement of the brakes and solves the problem of partial jams.

Interestingly, my Model S 2016 will solve this problem in the store tomorrow. I received an alert email saying that Tesla will replace the component for free in November due to a warranty extension. I noticed this because I first heard about this problem from the RichRebuilds YT channel.

Then you see, it is almost destined, in December, an alert popped up on my monitor, storage device degraded or something. The repair was booked even before the recall was announced, so I entered there before the crazy rush! Get a car and everything.

I noticed that only a few strange things happened in the car, especially some Spotify album tracks cached on the storage device were corrupted and unplayable, and the media player froze for about a minute. For whatever reason, the car did not download a new file to solve the problem. Apart from that, I have never been denied control of any aspect of the car, air conditioning, etc.

But rarely I have to do a two-finger salute to restart the center console, but even so, you can still drive the car. This must be done when the click sound is stopped when the turn indicator is active. I was driving, so I don’t know if these indicators are really effective. Either way, the quick restart will be up and running within a few minutes, while I am still driving it.

All in all, I can't complain.

> All in all, I can't complain. > When the turn indicator is active, it stops playing the click sound

This is such a basic (safe) feature that you have to do your best to justify your purchase and justify simple mistakes made by the company.

I mean, if it is a Toyota car worth $18,500 and the dashboard indicator fails due to a malfunction in the center console software, I will simply refuse it and return it to the dealer for a full refund. I will never touch that pile of bullshit-it will only cause trouble in the future; endless garage bills, and struggle with the company to solve problems, these problems should not be a problem in the first place.

But when it is a luxury car worth $100,000, do you deal with it?

It is not downloading new files to "fix it" because there is not enough space on the EMMC to place them. Hopefully their alternatives include disabling Linux operating system logging, which is not necessary for it to work, and disabling logging should have some benefits for the overall performance of the software.

Flash memory manufacturer, not BGA manufacturer. BGA refers to the packaging method of flash memory. In the plastic case, there is a "ball grid array" solder ball underneath.

Flash write durability is easy to understand. Most junior embedded engineers will tell you that frequently writing fast refreshed data to flash memory is a bad idea. Tesla has not been affected by the "sandbag" and has no recourse against the manufacturer because the product operates as designed.

what. Can't say that I was surprised.

I am afraid that all Tesla owners should prepare for more recalls in the coming years.

So they can pull a Ford; and then say, “We’re sorry we messed up here.” Give all affected car owners a new car at a reasonable discount, let the old car research wear/remanufacture and reuse parts, let They are far away from garbage dumps and secondary markets. Given the financial situation of the past few years, I bet that it might even be classified as a good investment. They will adjust the numbers as needed.

I want to know if Tesla has obtained their "wearable project" context from Microsoft. I say this because a friend discovered a major bug in Windows, and since they are a large company, they hope that Microsoft will update/fix the bug. Microsoft reviewed this issue and came back. They did not call it a bug, but called it a "feature", so they plan to take no action on this. Now, under some very conventional steps, "features" will cause a blue screen. My friend didn’t blink...just saying that their company is planning a press conference and inviting everyone so they can show and demonstrate this "feature"... Look, Microsoft decided to fix this bug/feature and said nothing .

There is nothing more appropriate than doing volunteer work to increase the profits of millionaires.

I can never understand some people

As a wear part, it should be easy to replace on site. It is impractical to require disassembly of the car dashboard to obtain "normal wear and tear items".

The era of disposable cars has arrived.

> The era of disposable cars has arrived.

Using glue batteries (the battery itself is a structural part of the frame), we can get there without worry. Just like how it is done in mobile phones today. The formula is known and tested.

(Lighter cars, faster production, safer, so this is a win-win for everyone, isn't it?)

Man, more than 8 years ago, I put SSDs in industrial PCs (they are not standard products you can order from our suppliers) and overconfigured them by 90%. The 8 GB partition on a 64 GB SSD is a bit like.

Hey, they still work!

All of this is meaningless, because the car will catch fire in less than 5 minutes after the collision. This is the basis of Tesla’s settlement out of court with money; when the car is damaged enough to damage the battery, it is almost impossible for rescuers to reach you before the thing catches fire, and without killing the car occupants , The flame is unstoppable.

eMMC / does / has a wear balance. What Tesla messed up was writing gigabytes of log data from vehicle sensors/cameras every day. It is not good to write such a large amount to eMMC. It must go wrong. Make such an n00b error.

As far as I know, they didn’t intend to write that much data, but someone messed up the recording settings, so it stored measurements much more frequently than expected (for example, recording the external temperature every 0.1 seconds instead of every Record the external temperature every 0.1 second) every ten minutes). So the real failure point is that their code review process did not find a problem with the software (that is, safety critical) that should run the car.

Tesla is using their fleet to collect large amounts of data for autonomous driving. Given that Elon Musk has always claimed to be able to use fully autonomous driving-I don't think they expected this level of data collection to last so long.

Space shuttles and O-rings, Toyota and throttle bodies that can malfunction and run at full speed, Tesla, and eMMC are just a small sample of how an object has such a high rating, but it is just another POS. Does anyone remember Zenith's sales slogan? The quality is entered before the name continues.

Be careful, Hakadi. Tesla will sue you because you dare to say anything negative about their stupid toy car, no matter how real it is. Heck, they even managed to shut up Jeremy Clarkson!

8GB? They didn't even make them 8 megabytes in size.

This happens when you are not using a real embedded system developer to write and manage embedded software/hardware. Generally speaking, PC developers never consider wear leveling and hardware life expectancy issues. A good embedded system engineer will not write software that burns flash memory like they do. So in my opinion, Tesla has excellent artificial intelligence software developers, but they are not qualified for embedded development. I hope they do the same at SpaceX.

Flash wear is the best way to start a flame war. You may even encounter nasty PMs whose level of abuse is ridiculous.

Linux people and their obsession with simplicity are more willing to rewrite entire huge text files every second and replace the SSD in 8 years, rather than adding a dependency on the code and using an appropriate database.

I am very happy that this world is not managed by programmers, otherwise you will need a new toilet every few years, and they will tell you not to use outdated hardware that is so cheap.

According to my billing rate. It is cheaper to replace it every other year than to clean it.

Well, it is a hard-wearing item, but if you want to weld/screw it in, then you'd better keep it in continuous use, or make it easy/cheap to replace. The former is best...

I have a pickup truck, and I need a new engine computer every year, about $2,000. Design flaws. Know more and do nothing. Again, they said it was normal. The entire industry is changing all the dead electrolytic caps because this thing is installed on the top of the engine and there are other things on the top.

This problem has been happening for several years. The computer has a removable storage that the Tesla Motors software can log into. It is detachable for easy diagnosis, especially when the car system is malfunctioning or not working at all.

The log that wears out the EMMC chip is the Linux operating system log that should be disabled. The software installed on the chip is mostly static, so the chip components it is on will not wear out. As more functions are added to the software, the log recording space is getting smaller and smaller, which makes the less and less space more utilized.

Place the EMMC on a small PCB with a surface mount connector (much like the Pentium III module used in early notebook computers) and a few mounting screws, which can be easily replaced by a Tesla service technician.

Tesla running out of SD cards for storing too much data is just another example of improper use of technology.

Ford recently promoted a new F150, it will automatically update the software for you, I asked why? I don't want a truck that updates software. I also don't want my TV to update its software. I also don't want Tesla that updates its software. (I don’t want my PC to be updated either) When my phone is updated or the calculator application on my phone needs to be updated, I just need to cry and wonder why? The same is true for cars that must store MB of data. Why? What might they use all this data for? Maybe they will lower your insurance based on the way they think they want you to drive. Before I buy your product, I would rather the software do a good job.

Sending all the data to the cloud is a worse idea. The car cannot drive because it is snowing and the clouds are scattered. Or because the SD card is full, Wipers will not run. Or wait a minute, because cloud services no longer exist, and can't sports cars anymore. Well, in order to drive this car, you must agree to its software update license and terms. Reject the clause, and then the car no longer runs.

My current car has been 15 years old. If the radar gets dirty, I will turn off the cruise control system and I will not buy similar ones again. It also replaced the communication board, antenna board, body control computer and numerous switches. Also, I short-circuited the sensor with the hood closed instead of replacing it. I have never owned one of them before, and I don’t need it now.

When I see the new features of driverless cars or self-driving cars, I want to know how they work in the winter in upstate New York. I laugh when Google Maps takes me through a logging yard or guides me around a seasonal bridge. I don’t laugh when Google Navigation tells me that the speed limit is 65 MPH and the highways are markedly less marked. Or when we are doing construction, the map is powerless.

One day, unless you plug it into WiFi and agree to the latest software update, we will not be able to drive the car and the toaster will not work. We all ask why?

This is like claiming that the rod end bearing is a wear part, please ignore the sand installed in the oil pan...

When the firmware had a checksum failure, Chevrolet replaced the computer in my truck. I don't know if the same memory is often written as non-volatile storage for records or error codes, or if it is just written for firmware at the factory, but it is based on flash memory and is not removable.

No manufacturer will store commercially sensitive data on easily removable media devices, even if encryption is in place. I am even more surprised that this device can be backed up (and JTAG is not fused) and restored instead of the BGA device used in the design. This is certainly not a wearable item, but frankly, Tesla (Elon) always does it to avoid messing up instead of going bankrupt. I must admit that I want to know why this is happening?

Why does a recording failure cause the car to enter limp mode? ? Bad design, stop buying TSLA stock.

Sending the defective code and then asking the customer to run in debug mode and collect the returned information in order to update the product's software model is not what I want to drive. For me, advertising your product with this feature is not a feature. Some people like their nightly build...I am happy for them and thank them for making the software more stable for people like me. Just don't include the nightly build built into my mom's car. Or mine.

This happens when you say that your car can run Cyberpunk 2077.

The basic principle is that customers should not replace computers with a value of more than $1,000 buried in the dashboard with "wear items." Unfair, poor engineering design, and not being called in for regular maintenance are inexcusable.

It indicates that the "new" generation of game consoles is now also using flash memory. The number of people who complain about the lack of storage space of these game consoles and the constant installation, uninstallation and reinstallation will become a problem in the future.

crazy. So in 2018, Tesla had a flash memory chip, and the replacement cost was more than $2,000. Is it now considered a vulnerable part? It's like Tesla thinks they are making mobile phones instead of cars and expects people to throw them away and buy a new phone every two years.

Absolutely agree. Elon Musk should take some out of his billions of bitcoins and pay for the repair himself, and then immediately redesign for better maintainability.

I made recommendations to Tesla on this matter in 2015. In 2017, they finally shut it down for 3 months and then restarted it. There isn't even any important thing, just basic Linux subsystem logging. Yes, newer cars, including 3/Y, still have this problem. They use 64G eMMC, so the wear time will be longer, but it certainly will. Since 2015, I have been solving this problem by moving the logs to the ramdisk. Yes, if it restarts, the logs will be lost, but no one looks at them anyway.

All key systems are constructed with redundancy in mind. Storage networks, like those that power AWS, have redundant power supplies, disks, network cards, etc. in each computer. If one fails, the other comes in, and the failed one can even be replaced without turning off the power. I believe the same principles can be applied to any critical system. Isn't a powerful car traveling at 140 miles per hour with other vehicles a critical system? Shouldn't there be at least one redundant flash memory on the board? It will only increase the price by a few dollars. This raises the question of whether they reduce redundancy everywhere in order to reduce costs.

Please be kind and respectful to help make the comment section great. (Comment Policy)

This website uses Akismet to reduce spam. Learn how to handle your comment data.

By using our website and services, you explicitly agree to the placement of our performance, functionality and advertising cookies. learn more