Apple - Is it safe to allow apps to access your photos?

Whether it is safe or not depends on whether you trust the app and the developer's privacy policy or not.

When you give access to an app to photos (either through the prompt that comes when the app tries to access photos or through Settings > Privacy > Photos), the app gets access to write new photos/images to your Camera Roll and to read all your photos on the device in an unencrypted form.

The date and time when each photo was taken is embedded within the photo. Plus, if you have enabled Location Services for the Camera app, then an app that has access to the Camera Roll can also find out where you took the photos.

How the app then treats the photos and the associated metadata (like date/time and location), if it uploads or sends them elsewhere, is left to the app. For example, once you give permission, iOS apps for popular services like Dropbox, Facebook, Flickr and Google+ can upload all your photos to their services. With iOS 7, these apps can also be allowed to do it in the background (when you're not actively running the app). Most users wouldn't even actively know which photos are being uploaded and when.

From the Data Management in iOS page of the developer documentation:

Similarly, your app can load, display, and edit photos from the Photo Library, as well as use the built-in camera to take new photos.

So far, Apple has not implemented any granular settings in iOS within Privacy > Photos to restrict read access to specific photos in the photo library.