Design a high accuracy ADC circuit (example of differential ADC LTC2348 for PSD signals)
Is there anything wrong I should change, be it in the schematic or the layout? This is to ensure the board meets the requirements.
I can't see the sensor, and the entire shielding schematic is probably not shown. but tying the shield to the ADC input is a little unusual. Typically shields are used to shunt current to ground and away from sensitive analog signals. I suppose a tying the shield to the negative analog input of the ADC is an attempt to get subtraction. With the filter only on the signal side common mode subtraction
Any electric field coupling into the shield should be shunted to ground, if ESD impinges on the shield there is a small chance it could be directed to the ADC, which is not good for ADC. It would be better to tie the shield to ground and create the lowest impedance pathway for the shield to ground (Unless you have some reference voltage on the shield, which from the PCB I would surmise the answer is no.)
A successful design will result in visualizing and managing grounding and return currents. Grounds are never zero volts, that is a concept that is needed when approaching the uV level. Cables are in the 100mΩ to 10mΩ resistance range. Lets say you have a grounding system that is 100mΩ and you have a switching load that is 3mV \$ V = IR \$ so \$ 3mA*100m\Omega = 300uV\$
A more than 3mA switching load would be seen without good grounding.
or \$ 3mA*10m\Omega = 30uV\$
So by lowering the resistance of the grounding system (cables connectors) will reduce common mode voltage noise by switching loads.
A happy analog system in the sub milivolt range is one where there are little to no switching loads on the ground system of the analog subsystem. This also means putting regulators and especially references close to the ADC (they will regulate the voltage to whatever their ground pin is, so even if you have a shifting ground, it will not cause problems if the ADC and the voltage reference see it)
As far as grounding goes, you do have ground loop if your running digital lines directly from the ADC to the processor, it would be better if you used digital isolators if possible between the two sections, especially if the motor board has a changing load (which the processor is a changing load in and of itself). If the grounding point is only between the two boards at the point you have shown any potential difference in the voltages of the ground system will result in a current. So it's best to isolate. And with the isolation, make sure the voltage regulators for the rails are also located next to the ADC. If your driving the analog sub system from a power supply, check the ripple on the power supply and how it will affect the analog subsystem, it's best to use voltage regulators to take care of ripple or noise from power supplies. The PSRR of the opamps (and ADC) will determine how much power supply noise gets into the signal.
If you want to do better than 300uV, make sure the ground plane has no currents traveling through the Analog section (like the blue line). Make sure currents from the shields return back to ground. When I say keep this zone free of return currents, don't have any loads that will send a return current back through the ADC section (current actually spreads out on the ground plane, then follows the path of lowest impedance (resistance at DC) back to the source (a power supply). For example, lets say the via at C35 was dumping a changing current of 10mA or more, and the power supply input was located at J19. The 10mA current would flow towards J19 and the ground pins (and caps ground pins) would see their grounds shift up and down, which is undesirable.
If your sensor is grounded this will result in problems and ground loops (Whatever SMA_CHx is connected to.
Other than that I can't comment on much without seeing more of the design.
IF your frequency range of the analog subsystem is DC to 600Hz, then by all means put a low pass filter on the ADC's at or near 600Hz. Filters decrease noise and a cap is a lot easier to add to a design than filtering digitally.