Why does an IllegalThreadStateException occur when Thread.start is called again

You could implement Runnable instead and use new Thread( $Runnable here ).start() or use a ExecutorService to reuse threads.

* It is never legal to start a thread more than once.
* In particular, a thread may not be restarted once it has completed
* execution.
*
* @exception IllegalThreadStateException  if the thread was already started
*/
public synchronized void start() {

In Android, document still mention that we will get IllegalThreadStateException if the thread was already started.
However for some device it will not throw this exception (tested on Kyocera 7.0). In some popular device like Samsung, HTC, it throw throw the exception normally

I answer here because the Android question is mark as duplicated to this question.

how could I kill or stop the thread each time, to get around this problem?

The answer is, you can't. Once started, a Thread may not be restarted. This is clearly documented in the javadoc for Thread. Instead, what you really want to do is new an instance of RemoveNonPrime each time you come around in your loop.

You have a few other problems in your code. First, you need to increment p before using it again:

for(int i = 0; i < cores; i++){
    t[i] = new removeNonPrime(f,p); //<--- BUG, always using p=2 means only multiples of 2 are cleared
}

Second, you might be multithreaded, but you aren't concurrent. The code you have basically only allows one thread to run at a time:

while(p <= (int)(Math.sqrt(N))){
    t[p%cores].start();//
    try{
        t[p%cores].join(); //<--- BUG, only the thread which was just started can be running now
    }catch(Exception e){}
    //get the next prime
    p++;
    while(p<=(int)(Math.sqrt(N))&&f[p]==0)p++;
}

Just my $0.02, but what you are trying to do might work, but the logic for selecting the next smallest prime will not always pick a prime, for example if one of the other threads hasn't processed that part of the array yet.

Here is an approach using an ExecutorService, there are some blanks (...) that you will have to fill in:

/* A queue to trick the executor into blocking until a Thread is available when offer is called */
public class SpecialSyncQueue<E> extends SynchronousQueue<E> {
    @Override
    public boolean offer(E e) {
        try {
            put(e);
            return true;
        } catch (InterruptedException ex) {
            Thread.currentThread().interrupt();
            return false;
        }
    }
}

ExecutorService executor = new ThreadPoolExecutor(cores, cores, new SpecialSyncQueue(), ...);
void pruneNonPrimes() {
    //...
    while(p <= (int)(Math.sqrt(N))) {
        executor.execute(new RemoveNonPrime(f, p));
        //get the next prime
        p++;
        while(p<=(int)(Math.sqrt(N))&&f[p]==0)p++;
    }


    //count primes
    int total = 0;
    System.out.println();

    for(int j=0; j<f.length;j++){
        if(f[j]!=0){
            total++;
        }
    }
    System.out.printf("Number of primes up to %d = %d",f.length,total);
}



class RemoveNonPrime extends Runnable {
    int k;
    int arr[];

    public RemoveNonPrime(int arr[], int k){
        this.arr = arr;
        this.k = k;
    }

    public void run(){
        int j = k*k;
        while(j<arr.length){
            if(arr[j]%k == 0)arr[j]=0;
            j+=k;
        }
    }
}