How do we know if a PCA plot is good? From online tutorials, the PC1 should not be very low or very high, a PC1 of 40-60% is good.
Do not pin a % of variability explained by PC1 to dataset quality, these do not necessarily correlate at all. Your PC1 may explain a relatively low or high percentage of the variability of the data, but that doesn't necessarily have anything to do with data quality.
Also, the samples should be neatly clustered into a single cluster.
In reality, it is not uncommon for there to be multiple groupings. The data is almost always more complicated than just x versus y, and there are usually more variables that differentiate the samples. This is especially true for human disease datasets. Sometimes, PC1 may not separate samples based on your condition of interest at all, but PC2, 3, 4, etc, might. Look at more than just PC1 and 2.
Can I still take forward these two datasets by picking the closely clustered samples?
You certainly can still use these datasets.
What is the minimum number of samples one should have to perform RNA-Seq in each group of a dataset?
This is a bit of a loaded question, and the answer depends on how much power you really want/need. There are a few papers that look at this with varying answers, like this one that recommends 6 for decent power and up to 12 if you really want to capture every difference. In practice, people typically shoot for at least 3.
Is there a statistical way to figure out which sub-clusters to choose for the next level of analysis?
There are all sorts of clustering algorithms to help break up samples, but first, you should take a look at other variables in the data that might explain why some samples tend to group together in a given PC. Depending on the data type, things like sex, tissue origin, age, read depth (scATAC is a prime example here, PC1 is always dominated by read depth differences), culture conditions, etc, can all play into how similar any set of samples appear in PCA.
