Evaluation Criteria for a File Transfer Server

"I liked the second one." "The second one was a data sheet." "Well, I liked the data sheet." That is how most shortlist meetings go. One candidate is judged on its demo, another on a PDF, a third on a colleague's memory of using it years ago at a different job. The result is a comparison of impressions, and impressions favor whoever presented last.

A criteria matrix fixes this: a written list of what will be judged, how each item will be verified, and how much each item counts. It is agreed before anyone scores anything. This article builds that matrix from the worksheet in the previous article. It covers nine categories of criteria, a way to verify each one, and honest guidance on weighting for three common situations. Those include the ones where a product like ours is the wrong answer. It is part of our Choosing a File Transfer Server series. It ends with a copyable matrix ready for the trial and the scoring that follow.

Gates First, Then Weights

Before the categories comes the disclosure this series repeats on purpose. Sysax sells a file transfer server, and this matrix is one we would expect to be scored against. We have written it so that it can eliminate us. If a criterion below seems suspiciously easy for a self-hosted Windows product to satisfy, check the weighting section and the section on where we are not the answer. Both exist to keep the matrix honest, and us with it.

A matrix has two kinds of rows, and confusing them is the most common way evaluations go wrong. A gate is a criterion a candidate must pass to be considered at all. It comes from a must row and is scored pass or fail, never on a scale. A weighted criterion comes from a should row. It is scored on a scale and multiplied by a weight that reflects how much it matters to you. Could rows are not in the matrix at all. They break ties at the end.

The order matters. Gates are applied first, to every candidate, and a candidate that fails any gate is removed before scoring begins. That is what stops a product brilliant at eight things from being bought despite failing the ninth thing you cannot live without. The diagram below shows the sequence.

The Nine Categories

Every criterion falls into one of nine categories. For each, the questions below are the ones that separate candidates in practice, with the way to verify the answer. A criterion you cannot verify is a criterion the vendor scores for you.

1. Protocol coverage

Does the server speak every protocol on your must list? Does it cover the variant your partners use: explicit and implicit FTPS, SFTP with the key types your partners hold, HTTPS for browser users? Can you disable the ones you do not want, so plain FTP cannot be quietly enabled later? Verify by connecting with the clients your partners actually use, not the vendor's. If your list includes AS2 for EDI partners, or direct delivery into cloud object storage, many general-purpose servers, ours included, do not cover those. In that case, this row becomes a gate that removes them.

2. Authentication options

Which methods exist, and can each account be limited to one? Directory logins for staff, public keys for partners, certificates for the few who need them, and something a browser user can manage. Check whether disabling an account in the directory disables it on the server immediately; offboarding that closes the account depends on it. Our comparison of authentication methods explains what each is for. If external users must sign in through a federated identity provider with single sign-on, that is a strength of portal-centric products. Traditional servers honestly score lower on it.

3. Security features

The features are cipher and protocol policy you can restrict, address allow and block lists, and lockout after repeated failures. They also include per-account confinement to a folder, a validated cryptographic mode if a regulation demands one, and banners that leak nothing. The detailed questions belong to the vendor security question set. Here the criterion is whether each control exists and can be verified in a trial by trying to break it. Our hardening program overview lists the controls a server should let you apply.

4. Logging depth

What is recorded (logins, failures, every file operation, every administrative change), with which fields, and where can it go: a file, a database, a central platform? Can you answer "what did this account do last Tuesday?" without a script? The what-to-log guide is the field list; evidence auditors accept tells you what the output must look like. Verify by pulling a real report during the trial, not by reading the data sheet.

5. Automation hooks

Can the server act when something happens: run a script on arrival, notify on failure, move a file after upload? Can accounts and folders be created from a script, so onboarding forty partners does not mean forty rounds of clicking? Products differ enormously here, and so do edition gates. If your rows describe orchestrating multi-step workflows across several systems with end-to-end visibility, that is the managed-transfer layer. The MFT assessment tells you whether you need it. A plain server scores low on that layer by design.

6. Administration experience

How long do the daily tasks take, and can the people you actually have do them? Onboarding a partner with a key, rotating that key, restricting an address, changing the passive port range, renewing a certificate, restoring the configuration to a fresh host. Time them in the trial; the trial article has the list. "Easy" depends on the team: a group that manages everything as code may score a text-file product higher than a console-driven one.

7. Performance

Does the server keep up with your peak on your hardware and links? That peak is the burst of partner uploads at the batch cutoff or the one large file that must finish before a deadline. Vendor throughput numbers are measured on the vendor's network with the vendor's file mix, on a good day. Our benchmarking series explains why they rarely transfer. If you genuinely need thousands of concurrent sessions across several sites with automatic failover, you are in the clustered category. For that need, single-server products should score low here whatever the brand.

8. Licensing model

What is the unit: a server, a core, a user, a connection, a protocol? Which features are gated behind editions, and does your must list touch any of them? What is the shape over time, one-time with maintenance or subscription? The criterion is fit with the budget shape from your requirements, never the amount. The amount is argued separately, in the one-page business case. The vendor question set covers how to get straight answers.

9. Support

Hours and time zones, who answers (an engineer or a ticket router), response versus resolution promises, how security advisories reach you, and how long a release stays supported. Verify by opening a real ticket during the trial and reading the terms rather than the brochure. If your policy demands vendor-run round-the-clock operations, a smaller vendor may not offer it, and that includes us; write the row and ask.

The Matrix, Ready to Copy

Here is the matrix in copyable form. Each row is marked G for a gate or W for weighted. The letter comes from your worksheet. So if a row is a must for you, mark it G even if it is W here. The verification column is the important one: it names the observation that produces the score.

EVALUATION MATRIX   G = gate (pass/fail)   W = weighted (0-4 scale)
                    VERIFY = the trial observation that produces the score

PROTOCOLS
 P1 W  speaks every listed protocol in the partner's variant   VERIFY: connect with partner-style clients
 P2 G  unwanted protocols can be disabled per server/account   VERIFY: disable FTP, attempt connection
 P3 W  passive range and external address configurable         VERIFY: FTPS from behind NAT succeeds
AUTHENTICATION
 A1 G  directory-integrated login for staff                     VERIFY: disable user in directory, login fails
 A2 G  public-key auth per partner account                      VERIFY: onboard test partner with key
 A3 W  method can be restricted per account (key only, etc.)   VERIFY: password login refused on key account
SECURITY
 S1 G  cipher/protocol policy restrictable                      VERIFY: weak cipher connection refused
 S2 W  address allow/block lists per account                    VERIFY: second address refused
 S3 W  lockout/throttling after repeated failures               VERIFY: scripted bad logins, observe lockout
 S4 G  per-account folder confinement                           VERIFY: attempt to list another partner's folder
LOGGING
 L1 G  logs login, failure, every file op, every admin change   VERIFY: pull a week for one account
 L2 W  log destinations: file, database, central platform       VERIFY: lines arrive centrally, fields intact
 L3 W  reports answer "who did what" without scripting          VERIFY: produce the report during trial
AUTOMATION
 U1 W  action on file arrival (script, notify, move)            VERIFY: drop file, watch action and its failure
 U2 W  accounts/folders creatable from a script                 VERIFY: create ten test accounts unattended
ADMINISTRATION
 D1 W  timed daily tasks within targets                         VERIFY: task timings sheet from the trial
 D2 W  configuration backup restores to a fresh host            VERIFY: restore in the trial, compare
 D3 W  runs as a service, returns after reboot unattended       VERIFY: reboot during a transfer
PERFORMANCE
 F1 W  meets the peak-window deadline on your hardware          VERIFY: replay peak with your file mix
LICENSING
 C1 W  unit and edition gates fit the must list and budget shape  VERIFY: vendor answers in writing
SUPPORT
 T1 W  ticket answered by someone who could fix it              VERIFY: open a real ticket during trial
 T2 W  advisory and release-support terms acceptable            VERIFY: written terms, not brochure

Two rules keep the matrix honest. First, every row's verification must be something your team performs. A vendor's written statement counts only for the licensing and support rows, where the statement is the thing being evaluated. Second, the matrix is finished and agreed before the first trial installation. Rows added afterwards need a written reason, exactly like the must list.

Weighting for Your Situation

Weights turn a list into a decision, and they are where evaluations get rigged, usually unconsciously, by whoever fills in the spreadsheet after the demos. I have done this myself; the column agreed with me completely, which should have been the warning. The defense is to set weights before the trial, from the requirements rather than from any product. A few rules apply: weights across the nine categories sum to one hundred. No category exceeds thirty. Gates are never weighted, because a gate is pass or fail. Two people set the weights independently, then reconcile the differences in writing.

There is no universal weighting, but three situations cover most readers. The table gives a defensible starting point for each; adjust from your worksheet, not from taste.

Read the columns as arguments, not answers. The small partner exchange, a dozen partners and two generalist admins, weights administration highest. That is because the admins' time is the scarcest resource and a product they cannot run will not be run well. The regulated enterprise weights logging and security because the evidence is the product. No auditor has ever asked how pleasant the console was. The internal automation hub, where most traffic is applications talking to applications, weights automation above everything. That is because a server that cannot trigger the next step is a folder with a login.

Notice what the weights do to a candidate like ours. Against the first column, a self-hosted Windows server with directory authentication and a console-driven admin model can score well, if the trial timings agree. Against the third, a product whose event triggers and scripted administration sit in a higher edition is scored on the edition you can actually buy. It may lose to a product built around automation. That is the weighting working as intended.

Bluewater Bank learned what weights set after the demos look like. Their architect filled in the column the afternoon after the second demo. Logging got five points, administration got thirty, and the favorite scored ninety-four. Internal audit asked one question: why did logging weigh five in a bank? Two people who had not seen the demos reset the weights. The favorite fell to third, and the bank bought the product that could produce a per-account activity report without a script. The architect's spreadsheet is still on the shared drive, in a folder called "superseded".

Where a Product Like Ours Is Not the Answer

An honest matrix produces eliminations a vendor would rather not list. Here they are from our side of the table, so you can recognize them from any vendor's side.

• Non-Windows estates. A Linux or mixed estate with no Windows operational skills should not adopt a Windows server for one workload. Products in our category fail this gate. The natural candidates are the platform's standard SSH server plus tooling, or a cross-platform commercial product.
• Cloud-native storage. If files must land in object storage, be addressed by presigned links, and be processed by cloud functions, a server with a disk is an extra hop. Our object storage article describes that shape. The products that fit it are built around the storage, not around a listener.
• Hosted-only policy. If nobody may run infrastructure, every self-hosted product is out. The self-hosted versus hosted comparison helps you decide whether that policy is really yours.
• Very large clustered deployments. Multiple active nodes behind a load balancer with shared state and site failover is a different product category with a different price shape and a different admin model.
• Protocols we do not speak. AS2, and anything not in the FTP, FTPS, SFTP, and HTTPS family, is a gate we fail.
• Full managed-transfer governance. Orchestration, end-to-end tracking across many systems, and a workflow designer are the MFT layers. A server plus an automation tool covers part of that, not all of it.

To be equally concrete in the other direction, here is one candidate against the authentication and logging rows. It is an example of the facts to collect for every product, not a verdict. Sysax Multi Server authenticates each account through built-in accounts, Windows and Active Directory accounts, or public keys. It restricts by IP allow and block lists and logs activity to a file and to a database. Event triggers and web-based administration are in its higher editions. That is exactly why the licensing row asks about edition gates. A trial that unlocks the full feature set lets you verify all of that yourself instead of taking our word. That is the standard for every vendor.

Criteria That Look Important but Aren't

A few things regularly sneak into matrices and distort them. Leave them out.

• Feature count. A longer data sheet is not a better product; features you will not use are surface area you will have to secure. Score the rows on your matrix and ignore the rest.
• Demo polish. A console that looks good in a scripted twenty-minute walkthrough tells you nothing about the ten-minute task you will do every week. Timed tasks in the trial replace this entirely.
• Compliance logos. A regulation's name on a brochure is a claim, not a control. Our guide to reading vendor security claims shows how to turn each logo into a verifiable question.
• Throughput headlines. Numbers measured on a vendor's lab network with large files say nothing about your many-small-files partner feed over a slow link. Test your own peak or skip the row.
• "Unlimited" anything. Unlimited users, connections, or servers usually has a boundary somewhere: in the edition, the support terms, or the hardware. Ask where.
• Market position. Rankings and quadrants measure vendors' size and marketing spend as much as fit. They are a source of shortlist names, not a criterion.

Vendor slide decks, ours among them, are optimistic documents. Nothing on that list is dishonest. Each is simply easy to present and hard to verify. The matrix only has room for things you can verify.

From Matrix to Trial

The finished matrix does two jobs at once. It is the scoring sheet the decision article will fill in. It is also the test plan for the trial. Every verification step in the right-hand column is a task on the trial calendar. That is not a coincidence. A criterion you cannot schedule into a trial is one you were never going to check.

Before moving on, run three sanity checks. Every gate should trace to a must row someone signed. The weights should carry the date and the two names that set them. And at least one row should be uncomfortable for your favorite candidate. That is because a matrix on which the favorite scores perfectly was written after the favorite was chosen. Someone will still have presented last; with the matrix in hand, it stops mattering. The next article turns the verification column into a trial that actually tests the server. For the commercial rows, the vendor question set gets answers in writing.

Frequently Asked Questions