I wouldn’t recommend it. The pattern was constructed with knit in mind, so using a different fabric, especially something not stretchy, could affect the fit and also the drape of the garment, (so the overall look could be different than you imagine). Especially if the garment is tight fitting, you would not only have to adjust to add a fastener, but you would also have to increase the pattern to add ease, since knit fabrics have negative ease. If you do decide to move forward, I would definitely recommend making a muslin and you may have to do several test runs.