Query without WHILE Loop

You need to use recursive query.

The 30days period is counted starting from prev(and no it is not possible to do it without recursion/quirky update/loop). That is why all the existing answer using only ROW_NUMBER failed.

WITH f AS (
  SELECT *, rn = ROW_NUMBER() OVER(PARTITION BY PatientId ORDER BY ApptDate) 
  FROM Appt1
), rec AS (
  SELECT Category = CAST('New' AS NVARCHAR(20)), ApptId, PatientId, ApptDate, rn, startDate = ApptDate
  FROM f
  WHERE rn = 1
  UNION ALL
  SELECT CAST(CASE WHEN DATEDIFF(DAY,  rec.startDate,f.ApptDate) <= 30 THEN N'FollowUp' ELSE N'New' END AS NVARCHAR(20)), 
         f.ApptId,f.PatientId,f.ApptDate, f.rn,
         CASE WHEN DATEDIFF(DAY, rec.startDate, f.ApptDate) <= 30 THEN rec.startDate ELSE f.ApptDate END
  FROM rec
  JOIN f
    ON rec.rn = f.rn - 1
   AND rec.PatientId = f.PatientId
)
SELECT ApptId, PatientId, ApptDate, Category
FROM rec
ORDER BY PatientId, ApptDate;  

db<>fiddle demo

Output:

+---------+------------+-------------+----------+
| ApptId  | PatientId  |  ApptDate   | Category |
+---------+------------+-------------+----------+
|      1  |       101  | 2020-01-05  | New      |
|      5  |       101  | 2020-01-25  | FollowUp |
|      6  |       101  | 2020-02-12  | New      |
|      7  |       101  | 2020-02-20  | FollowUp |
|      8  |       101  | 2020-03-30  | New      |
|      9  |       303  | 2020-01-28  | New      |
|     10  |       303  | 2020-02-02  | FollowUp |
|      2  |       505  | 2020-01-06  | New      |
|      3  |       505  | 2020-01-10  | FollowUp |
|      4  |       505  | 2020-01-20  | FollowUp |
+---------+------------+-------------+----------+

How it works:

1. f - get starting point(anchor - per every PatientId)
2. rec - recursibe part, if the difference between current value and prev is > 30 change the category and starting point, in context of PatientId
3. Main - display sorted resultset

---

Similar class:

Conditional SUM on Oracle - Capping a windowed function

Session window (Azure Stream Analytics)

Running Total until specific condition is true - Quirky update

---

Addendum

Do not ever use this code on production!

But another option, that is worth mentioning besides using cte, is to use temp table and update in "rounds"

It could be done in "single" round(quirky update):

CREATE TABLE Appt_temp (ApptID INT , PatientID INT, ApptDate DATE, Category NVARCHAR(10))

INSERT INTO Appt_temp(ApptId, PatientId, ApptDate)
SELECT ApptId, PatientId, ApptDate
FROM Appt1;

CREATE CLUSTERED INDEX Idx_appt ON Appt_temp(PatientID, ApptDate);

Query:

DECLARE @PatientId INT = 0,
        @PrevPatientId INT,
        @FirstApptDate DATE = NULL;

UPDATE Appt_temp
SET  @PrevPatientId = @PatientId
    ,@PatientId     = PatientID 
    ,@FirstApptDate = CASE WHEN @PrevPatientId <> @PatientId THEN ApptDate
                           WHEN DATEDIFF(DAY, @FirstApptDate, ApptDate)>30 THEN ApptDate
                           ELSE @FirstApptDate
                      END
    ,Category       = CASE WHEN @PrevPatientId <> @PatientId THEN 'New'
                           WHEN @FirstApptDate = ApptDate THEN 'New'
                           ELSE 'FollowUp' 
                      END
FROM Appt_temp WITH(INDEX(Idx_appt))
OPTION (MAXDOP 1);

SELECT * FROM  Appt_temp ORDER BY PatientId, ApptDate;

db<>fiddle Quirky update

You could do this with a recursive cte. You should first order by apptDate within each patient. That can be accomplished by a run-of-the-mill cte.

Then, in the anchor portion of your recursive cte, select the first ordering for each patient, mark the status as 'new', and also mark the apptDate as the date of the most recent 'new' record.

In the recursive portion of your recursive cte, increment to the next appointment, calculate the difference in days between the present appointment and the most recent 'new' appointment date. If it's greater than 30 days, mark it 'new' and reset the most recent new appointment date. Otherwise mark it as 'follow up' and just pass along the existing days since new appointment date.

Finallly, in the base query, just select the columns you want.

with orderings as (

    select       *, 
                 rn = row_number() over(
                     partition by patientId 
                     order by apptDate
                 ) 
    from         #appt1 a

),

markings as (

    select       apptId, 
                 patientId, 
                 apptDate, 
                 rn, 
                 type = convert(varchar(10),'new'),
                 dateOfNew = apptDate
    from         orderings 
    where        rn = 1

    union all
    select       o.apptId, o.patientId, o.apptDate, o.rn,
                 type = convert(varchar(10),iif(ap.daysSinceNew > 30, 'new', 'follow up')),
                 dateOfNew = iif(ap.daysSinceNew > 30, o.apptDate, m.dateOfNew)
    from         markings m
    join         orderings o 
                     on m.patientId = o.patientId 
                     and m.rn + 1 = o.rn
    cross apply  (select daysSinceNew = datediff(day, m.dateOfNew, o.apptDate)) ap

)

select    apptId, patientId, apptDate, type
from      markings
order by  patientId, rn;

I should mention that I initially deleted this answer because Abhijeet Khandagale's answer seemed to meet your needs with a simpler query (after reworking it a bit). But with your comment to him about your business requirement and your added sample data, I undeleted mine because believe this one meets your needs.