Can I determine whether the string can deserialize by newtonsoft?

    private static bool TryParseJSON(string json, out JObject jObject)
    {
        try
        {
            jObject = JObject.Parse(json);
            return true;
        }
        catch
        {
            jObject = null;
            return false;
        }
    }

worked just fine for my scenario

There is no TryParse in Json.Net as of the current release. If you don't have a known schema to validate against, and you don't want to use try...catch then your only other option that I can see is to attach an error handler to the serializer and use that as a means of detecting and/or handling errors. See "Error Handling" in the documentation.