Does PCI actually require card informations to be encrypted in memory?
TL;DR
Technically, PCI requires card holder data (CHD) to be encrypted both in transit and at rest. This may seem simple at first, but the reality is, it doesn't get very specific about delineating between rest on disk vs. volatile memory. If you want to be pedantic, volatile memory is still data at rest, but according to the FAQs, it's not explicitly required to be encrypted while in memory. Though it's good to point out that this is just the FAQs, not actually in the PCI DSS Standard itself. Here's a good article to read for more info.
Long Answer
The reason for in-memory encryption has to do with the memory-scraping attack vector.
PCI DSS does a good job of making sure credit card data in persistent storage is secure, however, such data in non-persistent storage -- such as files stored temporarily in memory -- can still be vulnerable to compromise, particularly via memory-scraping malware. Learn more about this threat. SOURCE
More info about what PCI requires can be found here. The TL;DR of the long answer is that if you do not encrypt the data at rest (on disk or in memory), make sure you have compensating controls to mitigate the risk of data loss.
If stored cardholder data cannot be encrypted, consult PCI DSS Appendix B: Compensating Controls and Appendix C: Compensating Controls Worksheet.
Cardholder data does not need to be encrypted in volatile memory (https://pcissc.secure.force.com/faq/articles/Frequently_Asked_Question/Should-cardholder-data-be-encrypted-while-in-memory) It should be noted that the ability for data written to volatile memory to persist onto disk can and will happen. The encryption of the pagefile or swap file can help prevent this.
Cardholder data must be protected when in transit over open, public networks [PCI DSS Req. 4.1] and when written to disk [PCI DSS Req. 3.4]. The requirements are not the same for each and examples are given within the PCI DSS. The Guidance column on the right of the standard is usually helpful as well.
The most recent version of the PCI DSS can be found here. https://www.pcisecuritystandards.org/document_library
To answer your second question first, ("I don't see reason behind it, if attacked can get to the values, he can get to encryption key as well, right?") the off-the-shelf RAM scraping malware simply knows how to identify track data in memory. It doesn't understand the memory map of your register process. It doesn't know your structures or fields. It doesn't try to find a char* buffer labeled "Track_Data". It just reads all the memory in your process and if it matches a pattern of 15 or 16 digits, it scrapes it up and sends it to the bad guy. The off-the-shelf malware certainly doesn't know what bytes make up an encryption key, so once your data is encrypted, the malware won't see it. (The typical malware is not very sophisticated, and certainly wouldn't recognize XOR or ROT-13 data, much less a block of AES-128 encrypted data.)
Consider the attackers have to be stealthy: if they sent 2GB RAM dumps from every cash register after every transaction, the network people would probably notice. They only send the smallest amount of data they need, and don't try to decrypt anything.
(Note that this is true only for the general purpose RAM scraping malware like Dexter and BlackPOS; if there is malware targeted at a specific application, it may be customized to understand the memory structures and to try to steal keys and/or encrypted data from specific addresses. It all depends on the attackers.)
--
As asked, the first part of your question almost describes a race condition: at what point is the track data considered "at rest"? Is it only when it lands on a hard disk, or can it be considered at rest while in memory waiting to be encrypted?
But now you have to keep going, and get more detailed. Is it at rest when it lands in the USB device driver's buffers? Is it resting when a Windows message is posted to notify your process that new track data has been input? Is it resting when the buffer is sent to the encryption algorithm? Is it at rest when the application parses the track to recover the cardholder name, service code, PAN, CVV, and expiration date? Is it at rest when it strips the last 4 digits for the receipt? Is it at rest if the application is processing the account number in any way prior to encrypting; perhaps validating a check digit, or determining if it's a Visa or other type of credit card? Is it at rest when the authorization message is being constructed? None of those activities seem like the data is resting very much, and many of those activities are common functions in POS systems. In total, all that processing can take a few milliseconds or more to execute.
Now let's consider the real-world attackers. The malware used in some of the biggest-name breaches was RAM scraping malware, and it is extremely aggressive. It can sweep through the register's memory hundreds of times per second, and trigger on the briefest glimpse of PAN or track data. It can capture track data arriving in the USB buffers, in the windows message buffers before the application is notified, while being taken apart in the parsing routines, or even as the encryption routines are being invoked. Even if you perfectly comply with the letter of the PCI-DSS laws and encrypt it as soon as your application sees it, you're still vulnerable to a breach if some member of this family of malware makes it onto your cash registers.
Instead of focusing on the exact interpretation of the PCI-DSS, it's better to avoid the situation entirely by removing as many systems as possible from PCI scope. If you can, get the track data out of the cash register, and into a separate payment terminal that can encrypt the data before sending it to your cash register.
If you move encryption to a separate device located outside of the cash register environment, then the register never needs to receive cleartext PAN or track data. Any malware that finds its way onto your registers will find no track data to scrape. And a payment terminal will have a much smaller attack surface than a full-blown Windows-based cash register application.