(CNN) -

On a day when so many Americans felt joy, peace and life-long inspiration from the March on Washington, then-Attorney General Bobby Kennedy was so nervous he could barely sit down.

"Pacing around the room," described Jack Rosenthal, Kennedy's deputy press secretary on Aug. 28, 1963. "The attorney general (was) off and on the phone, talking I presume to march organizers or to the White House."

Rosenthal was with Kennedy inside the "command center" that Justice Department officials used to monitor the march inside the Justice Department headquarters.

This week marks 50 years since the march and the Rev. Martin Luther King, Jr.'s pivotal "I Have a Dream" speech. While the event has been analyzed by countless historians, the Kennedy administration's relationship with the march wasn't so easily understood.

"The Kennedys were almost morbidly afraid of this march. They understood there'd been nothing like it," said Rep. Eleanor Holmes Norton, D-District of Columbia, who helped plan the march 50 years ago.

Yet President John F. Kennedy and his attorney general brother knew the march had to be successful, so they assigned a small staff of Justice Department officials to help.

That help was so considerable that the head of the Civil Rights division of the Justice Department at the time, Burke Marshall, would later say about the march: "The person who organized it, as a matter of fact, was the attorney general."

Critics of the march accused the Kennedy administration of being too involved.

After referring to it as the "Farce on Washington," Malcolm X would write in his autobiography, "there wasn't a single logistics aspect uncontrolled. The marchers had been instructed to bring no signs. ... They had been told how to arrive, when, where to arrive, where to assemble, when to start marching, the route to march. ... Yes, I was there. I observed that circus."

So, how did the Kennedy administration go from being against the march to being so involved that they were accused of controlling it? Did President Kennedy and his brother help, hinder or co-opt the march?

The answer lies somewhere in between.

'He thought it would be chaos'

When President Kennedy first heard of a proposed march on Washington, he wasn't exactly thrilled.

It was June 1963 and Kennedy was meeting with civil rights leaders at the White House, including 23-year-old John Lewis, who had just been elected to lead the Student Nonviolent Coordinating Committee.

"It was a very moving meeting," Lewis said. "The president was deeply concerned about what was happening in the American South. So the president wanted to know what could be done."

Then A. Philip Randolph, the elderly labor leader revered by many in the civil rights movement, announced to the president that there would be a march on Washington.

"You can tell by the very body language of President Kennedy ... he started moving and twisting in his chair. And his facial expression -- he just thought it would be chaos," recalled Lewis.

"And the president sort of said, 'Well, I think we're gonna have some problems.'"

Despite being in favor of civil rights, Kennedy's reason for opposing the march was simple:

"The Kennedy administration was afraid that if there was violence on the march, it would mean that the Civil Rights Act, which John F. Kennedy had just introduced, would never get passed," said march planner Rachelle Horowitz.

Kennedy explained his concerns to the civil rights leaders in his office.

"We want success in the Congress, not just a big show at the Capitol," Kennedy is quoted as saying. "Some of these people are looking for an excuse to be against us, and I don't want to give any of them a chance to say, 'Yes I'm for the bill but I am damned if I will vote for it a the point of a gun.'

"The only effect is to create an atmosphere of intimidation - and this may give some members of Congress an out."

But the civil rights leaders would not be discouraged.

The March on Washington planning committee set up an office on 130th Street in Harlem in summer 1963 and began the arduous task of trying to contact, recruit and deliver thousands of people to attend the march.