Line the 8-inch square baking dish with parchment paper, leaving some overhang for easy removal later. Set aside.
In a medium saucepan over medium heat, combine the butter, butterscotch chips, and sweetened condensed milk. Stir constantly until the mixture is smooth and melted.
Once melted, remove the saucepan from heat and stir in the pumpkin puree, vanilla extract, ground cinnamon, ground nutmeg, ground ginger, and salt. Mix until well combined and smooth.
If using nuts, fold in the chopped walnuts or pecans at this stage, ensuring even distribution.
Pour the fudge mixture into the prepared baking dish, smoothing the top with a rubber spatula. Tap the dish lightly on the counter to release any air bubbles.
Allow the fudge to cool at room temperature for about 30 minutes, then transfer it to the refrigerator for an additional 2 hours, or until it is completely set.
Once set, lift the fudge out of the baking dish using the parchment paper overhang. Cut into 12 even squares and serve.