how set primeng data table columns auto fit?

You need to use 2 properties [resizableColumns]="true" [autoLayout]="true" as defined below inside the <p-table> tag

<p-table [resizableColumns]="true" [autoLayout]="true">

Why it happens? If you inspect it, primeng would by default add the following code from the primeng.min.css file

.ui-table table {
    border-collapse: collapse;
    width: 100%;
    table-layout: fixed;
}

You need to change the table-layout property , for this you need to use the [autoLayout]="true". There is another way also in which you can override this by using :host but i would not recommend it as autoLayout is already available to use.

For resizing, you need to use [resizableColumns]="true" in tag

For scrolling, you need to use [scrollable]="true" scrollHeight="300px" in tag

I have used following code. Changed in SCSS set table-layout:auto !important; .html

<p-dataTable [value]="Dataset"  [style]="{'width':'100%','overflow':'auto!important'}" 
      [responsive]="true"  [rows]="20" 
       [resizableColumns]="true" 
       columnResizeMode="expand" 
       [immutable]=false
       [paginator]="true" [rowsPerPageOptions]="[10,15,20,25]"
      appendTo="body" #dt>
      <p-column  styleClass="checkbox ui-resizable-column" [style]="{'width': 'auto'}" selectionMode="multiple">
      </p-column>  
      <p-column *ngFor="let col of cols;let j=index;" [style]="{'width':'auto','display':col.display} " [field]="col.field" [header]="col.header"
        [sortable]="true" [filter]="true" filterPlaceholder="Search" (mouseleave)="hoveredIndex=null" filterPlaceholder="Search"
        appendTo="body">
        <ng-template let-row="rowData" let-i="rowIndex" pTemplate="body">
          <div [pTooltip]="row[col.field]" [id]="col.field"></div>
             <!-- set String  -->
            <span (click)="test(dt)"  style="text-align:left;" *ngIf="col.datatype!='int' && col.datatype!='float'">
              {{row[col.field]}}
            </span>
             <!-- set int  -->
            <span  (click)="test(dt)"  style="text-align:top;float: top;padding-top: 4px !important;" *ngIf="col.datatype=='int' || col.datatype=='float' ">
              {{row[col.field]}}
            </span>
        </ng-template>
      </p-column>
    </p-dataTable> 

---

.scss

@import "src/app/Themes/colorVariables";  //datatable ui
    //Updated row
    ::ng-deep .ui-datatable tbody > tr.ng-star-inserted.ui-widget-updated-row{
        background-color:$updated-row-color;
    } 
    ::ng-deep .ui-datatable tbody > tr>td.ui-widget-deleted-row-checkbox  .ui-chkbox{
        display: none;
        }
    //Deleted row
    ::ng-deep .ui-datatable tbody > tr.ng-star-inserted.ui-widget-deleted-row{
        background-color:$deleted-row-color;
    } 

    ::ng-deep  .ui-datatable table
    {
        table-layout:auto !important;
        overflow-y: scroll !important; 
    }
    ::ng-deep .ui-datatable-tablewrapper {
        overflow-y: scroll !important; 
        width: auto !important;
    }
    ::ng-deep .ui-datatable-resizable {
        padding-bottom: 1px;
        /* overflow: auto; */
        width: auto !important;
    }

    ::ng-deep .ui-datatable-scrollable-view .ui-datatable-scrollable-body {
       // min-height: 300px;
        border: 1px solid #D5D5D5;
    }

    ::ng-deep .ui-datatable tbody > tr.ui-widget-content.ui-state-highlight{
        background-color: darkkhaki !important; 
    }
    ::ng-deep a.ui-state-highlight , .ui-state-highlight{
        background-color: rgb(64, 153, 83);
        color: black;
    }
    .hoverAction:hover{
        background-color: seagreen;
        color: black;
    }
    ::ng-deep .ui-confirmdialog-message {
        white-space: pre-line;
    }
    ::ng-deep  .ui-datatable tr.ui-datatable-even:hover
    {  background: #78BCFF;
    }
    ::ng-deep .ui-datatable tbody > tr.ui-widget-content.ui-datatable-odd:hover {
        background: #78BCFF;
    }

    .ui-datatable .ui-datatable-thead>tr>th, .ui-datatable .ui-datatable-tfoot>tr>td, .ui-datatable .ui-datatable-data>tr>td {
        border-color: inherit;
        -webkit-box-sizing: border-box;
        box-sizing: border-box;
        padding: .25em .5em;
        border-width: 1px;
        flex: 2;
        //width:auto !important;
        min-height: 8px;
        min-width: auto !important;
        max-width: 300px !important;
        font-size: 12px;
        padding: 0px !important;
        padding-left: 4px !important;
        color: black;
        text-transform: capitalize;
        white-space: nowrap;
        overflow: hidden;
         display: table-cell;
        text-overflow: ellipsis !important;
        word-wrap: break-word !important;
        /* font-size: 11px; */
        font-family: $default-font-family;
        border-width: 1px;
        border-style: solid;
    }