We introduce the alternating Schur algebra AS(F)(n, d) as the commutant of the action of the alternating group A(d) on the d-fold tensor power of an n-dimensional F-vector space. When F has characteristic different from 2, we give a basis of AS(F)(n, d) in terms of bipartite graphs, and a graphical interpretation of the structure constants. We introduce the abstract Koszul duality functor on modules for the even part of any Z/2Z-graded algebra. The algebra AS(F)(n, d) is Z/2Z-graded, having the classical Schur algebra S-F(n, d) as its even part. This leads to an approach to Koszul duality for S-F(n, d)-modules that is amenable to combinatorial methods. We characterize the category of AS(F)(n, d)-modules in terms of S-F(n, d)-modules and their Koszul duals. We use the graphical basis of AS(F)(n, d) to study the dependence of the behavior of derived Koszul duality on n and d.